{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Orchestrating Jobs, Model Registration, Continuous Deployment, and Lineage Tracking with Amazon SageMaker\n",
    "\n",
    "Amazon SageMaker offers Machine Learning application developers and Machine Learning operations engineers the ability to orchestrate SageMaker jobs and author reproducible Machine Learning pipelines, deploy custom-build models for inference in real-time with low latency or offline inferences with Batch Transform, and track lineage of artifacts. You can institute sound operational practices in deploying and monitoring production workflows, deployment of model artifacts, and track artifact lineage through a simple interface, adhering to safety and best-practice paradigmsfor Machine Learning application development.\n",
    "\n",
    "The SageMaker Workflow service supports a SageMaker Machine Learning Pipeline Domain Specific Language (DSL), which is a declarative Json specification. This DSL defines a Directed Acyclic Graph (DAG) of pipeline parameters and SageMaker job steps. The SageMaker Python Software Developer Kit (SDK) streamlines the generation of the pipeline DSL using constructs that are already familiar to engineers and scientists alike.\n",
    "\n",
    "The SageMaker Model Registry is where trained models are stored, versioned, and managed. Data Scientists and Machine Learning Engineers can compare model versions, approve models for deployment, and deploy models from different AWS accounts, all from a single Model Registry. SageMaker enables customers to follow the best practices with ML Ops and getting started right. Customers are able to standup a full ML Ops end-to-end system with a single API call.\n",
    "\n",
    "And the SageMaker Lineage service makes it easy to track all the artifacts created in a SageMaker Machine Learning Pipeline from start to finish."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use Case: Fine-Tune a BERT Model and Create a Text Classifier\n",
    "\n",
    "We have chosen the [Amazon Customer Reviews Dataset](https://s3.amazonaws.com/amazon-reviews-pds/readme.html) as our main dataset.\n",
    "\n",
    "- `marketplace`: 2-letter country code (in this case all \"US\").\n",
    "- `customer_id`: Random identifier that can be used to aggregate reviews written by a single author.\n",
    "- `review_id`: A unique ID for the review.\n",
    "- `product_id`: The Amazon Standard Identification Number (ASIN).  `http://www.amazon.com/dp/<ASIN>` links to the product's detail page.\n",
    "- `product_parent`: The parent of that ASIN.  Multiple ASINs (color or format variations of the same product) can roll up into a single parent.\n",
    "- `product_title`: Title description of the product.\n",
    "- `product_category`: Broad product category that can be used to group reviews (in this case digital videos).\n",
    "- `star_rating`: The review's rating (1 to 5 stars).\n",
    "- `helpful_votes`: Number of helpful votes for the review.\n",
    "- `total_votes`: Number of total votes the review received.\n",
    "- `vine`: Was the review written as part of the [Vine](https://www.amazon.com/gp/vine/help) program?\n",
    "- `verified_purchase`: Was the review from a verified purchase?\n",
    "- `review_headline`: The title of the review itself.\n",
    "- `review_body`: The text of the review.\n",
    "- `review_date`: The date the review was written.\n",
    "\n",
    "![BERT Training](img/bert_training.png)\n",
    "\n",
    "BERT’s attention mechanism is called a Transformer. This is, not coincidentally, the name of the popular BERT Python library, “Transformers,” maintained by a company called [HuggingFace](https://github.com/huggingface/transformers). We will use a variant of BERT called [DistilBert](https://arxiv.org/pdf/1910.01108.pdf) which requires less memory and compute, but maintains very good accuracy on our dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SageMaker Pipelines\n",
    "\n",
    "Amazon SageMaker Pipelines support the following:\n",
    "\n",
    "* Pipelines - A Directed Acyclic Graph of steps and conditions to orchestrate SageMaker jobs and resource creation.\n",
    "* Processing Job steps - A simplified, managed experience on SageMaker to run data processing workloads, such as feature engineering, data validation, model evaluation, and model interpretation.\n",
    "* Training Job steps - An iterative process that teaches a model to make predictions by presenting examples from a training dataset.\n",
    "* Conditional step execution - Provides conditional execution of branches in a pipeline.\n",
    "* Registering Models - Creates a model package resource in the Model Registry that can be used to create deployable models in Amazon SageMaker.\n",
    "* Parametrized Pipeline executions - Allows pipeline executions to vary by supplied parameters.\n",
    "* Transform Job steps - A batch transform to preprocess datasets to remove noise or bias that interferes with training or inference from your dataset, get inferences from large datasets, and run inference when you don't need a persistent endpoint.\n",
    "\n",
    "\n",
    "## Our BERT Pipeline\n",
    "\n",
    "In the Processing Step, we perform Feature Engineering to create BERT embeddings from the `reviews_body` text using the pre-trained BERT model, and split the dataset into train, validation and test files. To optimize for Tensorflow training, we saved the files in TFRecord format. \n",
    "\n",
    "In the Training Step, we fine-tune the BERT model to our Customer Reviews Dataset and add a new classification layer to predict the `star_rating` for a given `review_body`.\n",
    "\n",
    "In the Evaluation Step, we take the trained model and a test dataset as input, and produce a JSON file containing classification evaluation metrics.\n",
    "\n",
    "In the Condition Step, we decide whether to register this model if the accuracy of the model, as determined by our evaluation step exceeded some value. \n",
    "\n",
    "\n",
    "\n",
    "![](./img/bert_sagemaker_pipeline.png)\n",
    "\n",
    "## SageMaker Model Registry\n",
    "\n",
    "Amazon SageMaker Model Registry supports the following:\n",
    "\n",
    "* Catalog models after the training step - data scientists run tens to thousands of experiments and may select a small set of models as candidates for production.\n",
    "* Manage model versions - data scientists can register new models which will be automatically versioned in the model registry.\n",
    "* Compare models - data scientists can run model evaluation steps in the pipeline and generate model metrics (e.g. accuracy metrics and bias metrics) which are recorded in the Model Registry and can be used to compare model versions.\n",
    "* Approve models - data scientists can mark model versions as “approved” or “rejected”. Alternately, the pipeline can also automate the model approvals. If there is a deployment pipeline associated with a Model and a live endpoint, then the model version is propagated in production. \n",
    "* Deploy models in different AWS accounts - models in the Model Registry support resource sharing across accounts which enables models built in data scientist accounts to be deployed in different pre-production and production accounts.\n",
    "\n",
    "## SageMaker Lineage\n",
    "\n",
    "Amazon SageMaker Lineage supports the following:\n",
    "\n",
    "* Automatically tracks all the artifacts created in a machine learning workflow from start to finish.  Modeled as a directed graph like structure.\n",
    "* Explore the lineage artifacts with easy to use SDK methods.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook Overview\n",
    "\n",
    "This notebook shows how to:\n",
    "\n",
    "### SageMaker Workflows\n",
    "\n",
    "* Define a set of Workflow Parameters that can be used to parametrize a Workflow Pipeline\n",
    "* Define a Processing step that performs cleaning and feature engineering, splitting the input data into train and test data sets\n",
    "* Define a Training step that trains a model on the pre-processed train data set\n",
    "* Define a Processing step that evaluates the trained model's performance on the test data set\n",
    "* Define a Register Model step that creates a model package from the estimator and model artifacts used in training\n",
    "* Define a Conditional step that measures a condition based on output from prior steps and conditionally executes the Register Model step\n",
    "* Define and create a Pipeline in a Workflow DAG, with the defined parameters and steps defined\n",
    "* Start a Pipeline execution and wait for execution to complete\n",
    "\n",
    "### SageMaker Model Registry\n",
    "\n",
    "* Create a SageMaker Project based on the Model Package Group name from the pipeline execution defined before\n",
    "\n",
    "\n",
    "### SageMaker Lineage\n",
    "\n",
    "Amazon SageMaker Lineage supports the following:\n",
    "\n",
    "* Provide the inputs and outputs of SageMaker job artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A SageMaker Workflow\n",
    "\n",
    "The pipeline that we create follows a typical Machine Learning Application pattern of pre-processing, training, evaluation, and model registration:\n",
    "\n",
    "![A typical ML Application pipeline](img/pipeline-full.png)\n",
    "\n",
    "### Create SageMaker Clients and Session\n",
    "\n",
    "First, we create a new SageMaker Session in the current AWS region. We also acquire the role arn for the session.\n",
    "\n",
    "This role arn should be the execution role arn that you set up in the Prerequisites section of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from botocore.exceptions import ClientError\n",
    "\n",
    "import os\n",
    "import sagemaker\n",
    "import logging\n",
    "import boto3\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "\n",
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "sm = boto3.Session().client(service_name='sagemaker', region_name=region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Track Pipeline In `Experiment`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "timestamp = int(time.time())\n",
    "\n",
    "pipeline_name = 'BERT-pipeline-{}'.format(timestamp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'smexperiments'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-a2cae71afffd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0msmexperiments\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexperiment\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mExperiment\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m experiment = Experiment.create(\n\u001b[1;32m      5\u001b[0m                 \u001b[0mexperiment_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpipeline_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'smexperiments'"
     ]
    }
   ],
   "source": [
    "import time\n",
    "from smexperiments.experiment import Experiment\n",
    "\n",
    "experiment = Experiment.create(\n",
    "                experiment_name=pipeline_name,\n",
    "                description='Amazon Customer Reviews BERT Pipeline Experiment', \n",
    "                sagemaker_boto_client=sm)\n",
    "\n",
    "experiment_name = experiment.experiment_name\n",
    "print('Experiment name: {}'.format(experiment_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create the `Trial`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from smexperiments.trial import Trial\n",
    "\n",
    "trial = Trial.create(trial_name='trial-{}'.format(timestamp),\n",
    "                     experiment_name=experiment_name,\n",
    "                     sagemaker_boto_client=sm)\n",
    "\n",
    "trial_name = trial.trial_name\n",
    "print('Trial name: {}'.format(trial_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create the `Experiment Config`'s for Each Step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_config_prepare = {\n",
    "    'ExperimentName': experiment_name,\n",
    "    'TrialName': trial_name,\n",
    "    'TrialComponentDisplayName': 'prepare'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_config_train = {\n",
    "    'ExperimentName': experiment_name,\n",
    "    'TrialName': trial_name,\n",
    "    'TrialComponentDisplayName': 'train'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_config_evaluate = {\n",
    "    'ExperimentName': experiment_name,\n",
    "    'TrialName': trial_name,\n",
    "    'TrialComponentDisplayName': 'evaluate'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_config_model_register = {\n",
    "    'ExperimentName': experiment_name,\n",
    "    'TrialName': trial_name,\n",
    "    'TrialComponentDisplayName': 'model_register'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Specify the Raw Inputs S3 Location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_input_data_s3_uri = 's3://{}/amazon-reviews-pds/tsv/'.format(bucket)\n",
    "print(raw_input_data_s3_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 ls $raw_input_data_s3_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define Parameters to Parametrize Pipeline Execution\n",
    "\n",
    "We define Workflow Parameters by which we can parametrize our Pipeline and vary the values injected and used in Pipeline executions and schedules without having to modify the Pipeline definition.\n",
    "\n",
    "The supported parameter types include:\n",
    "\n",
    "* `ParameterString` - representing a `str` Python type\n",
    "* `ParameterInteger` - representing an `int` Python type\n",
    "* `ParameterFloat` - representing a `float` Python type\n",
    "\n",
    "These parameters support providing a default value, which can be overridden on pipeline execution. The default value specified should be an instance of the type of the parameter.\n",
    "\n",
    "The parameters defined in this workflow below include:\n",
    "\n",
    "* `processing_instance_type` - The `ml.*` instance type of the processing job.\n",
    "* `processing_instance_count` - The instance count of the processing job. For illustrative purposes only: 1 is the only value that makes sense here.\n",
    "* `training_instance_type` - The `ml.*` instance type of the training job.\n",
    "* `model_approval_status` - What approval status to register the trained model with for CI/CD purposes. Defaults to \"PendingManualApproval\". (NOTE: not available in service yet)\n",
    "* `input_data` - The URL location of the input data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pipeline Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.workflow.parameters import (\n",
    "    ParameterInteger,\n",
    "    ParameterString,\n",
    "    ParameterFloat,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experiment Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "exp_name = ParameterString(\n",
    "    name=\"ExperimentName\",\n",
    "    default_value=experiment_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing Step Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data = ParameterString(\n",
    "    name=\"InputData\",\n",
    "    default_value=raw_input_data_s3_uri,\n",
    ")\n",
    "\n",
    "processing_instance_count = ParameterInteger(\n",
    "    name=\"ProcessingInstanceCount\",\n",
    "    default_value=1\n",
    ")\n",
    "\n",
    "processing_instance_type = ParameterString(\n",
    "    name=\"ProcessingInstanceType\",\n",
    "    default_value=\"ml.c5.2xlarge\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_seq_length = ParameterInteger(\n",
    "    name=\"MaxSeqLength\",\n",
    "    default_value=64,\n",
    ")\n",
    "\n",
    "balance_dataset = ParameterString(\n",
    "    name=\"BalanceDataset\",\n",
    "    default_value=\"True\",\n",
    ")\n",
    "    \n",
    "train_split_percentage = ParameterFloat(\n",
    "    name=\"TrainSplitPercentage\",\n",
    "    default_value=0.90,\n",
    ")\n",
    "\n",
    "validation_split_percentage = ParameterFloat(\n",
    "    name=\"ValidationSplitPercentage\",\n",
    "    default_value=0.05,\n",
    ")\n",
    "\n",
    "test_split_percentage = ParameterFloat(\n",
    "    name=\"TestSplitPercentage\",\n",
    "    default_value=0.05,\n",
    ")\n",
    "\n",
    "feature_store_offline_prefix = ParameterString(\n",
    "    name=\"FeatureStoreOfflinePrefix\",\n",
    "    default_value=\"reviews-feature-store-\" + str(timestamp),\n",
    ")\n",
    "\n",
    "feature_group_name = ParameterString(\n",
    "    name=\"FeatureGroupName\",\n",
    "    default_value=\"reviews-feature-group-\" + str(timestamp)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pygmentize ./preprocess-scikit-text-to-bert-feature-store.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create an instance of an `SKLearnProcessor` processor and we use that in our `ProcessingStep`.\n",
    "\n",
    "We also specify the `framework_version` we will use throughout.\n",
    "\n",
    "Note the `processing_instance_type` and `processing_instance_count` parameters that used by the processor instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "\n",
    "processor = SKLearnProcessor(framework_version='0.23-1',\n",
    "                             role=role,\n",
    "                             instance_type=processing_instance_type,\n",
    "                             instance_count=processing_instance_count,\n",
    "                             env={'AWS_DEFAULT_REGION': region},                             \n",
    "                             # max_runtime_in_seconds=7200\n",
    "                            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "from sagemaker.workflow.steps import ProcessingStep\n",
    "\n",
    "processing_inputs=[\n",
    "        ProcessingInput(\n",
    "            input_name='raw-input-data',\n",
    "            source=input_data,\n",
    "            destination='/opt/ml/processing/input/data/',\n",
    "            s3_data_distribution_type='ShardedByS3Key'\n",
    "        )\n",
    "]\n",
    "\n",
    "processing_outputs=[\n",
    "        ProcessingOutput(output_name='bert-train',\n",
    "                         s3_upload_mode='EndOfJob',\n",
    "                         source='/opt/ml/processing/output/bert/train',\n",
    "                        ),\n",
    "        ProcessingOutput(output_name='bert-validation',\n",
    "                         s3_upload_mode='EndOfJob',                         \n",
    "                         source='/opt/ml/processing/output/bert/validation',\n",
    "                        ),\n",
    "        ProcessingOutput(output_name='bert-test',\n",
    "                         s3_upload_mode='EndOfJob',\n",
    "                         source='/opt/ml/processing/output/bert/test',\n",
    "                        ),\n",
    "]        \n",
    "\n",
    "processing_step = ProcessingStep(\n",
    "    name='Processing', \n",
    "    code='preprocess-scikit-text-to-bert-feature-store.py',\n",
    "    processor=processor,\n",
    "    inputs=processing_inputs,\n",
    "    outputs=processing_outputs,\n",
    "    job_arguments=['--train-split-percentage', str(train_split_percentage.default_value),                   \n",
    "                   '--validation-split-percentage', str(validation_split_percentage.default_value),\n",
    "                   '--test-split-percentage', str(test_split_percentage.default_value),\n",
    "                   '--max-seq-length', str(max_seq_length.default_value),\n",
    "                   '--balance-dataset', str(balance_dataset.default_value),\n",
    "                   '--feature-store-offline-prefix', str(feature_store_offline_prefix.default_value),\n",
    "                   '--feature-group-name', str(feature_group_name.default_value)\n",
    "                  ]\n",
    ")        \n",
    "\n",
    "print(processing_step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Define a Processing Step for Feature Engineering](img/pipeline-2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we use the processor instance to construct a `ProcessingStep`, along with the input and output channels and the code that will be executed when the pipeline invokes pipeline execution. This is very similar to a processor instance's `run` method, for those familiar with the existing Python SDK.\n",
    "\n",
    "Note the `input_data` parameters passed into `ProcessingStep` as the input data of the step itself. This input data will be used by the processor instance when it is run.\n",
    "\n",
    "Also, take note the `\"bert-train\"`, `\"bert-validation\"` and `\"bert-test\"` named channels specified in the output configuration for the processing job. Such step `Properties` can be used in subsequent steps and will resolve to their runtime values at execution. In particular, we'll call out this usage when we define our training step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train Step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_instance_type = ParameterString(\n",
    "    name=\"TrainingInstanceType\",\n",
    "    default_value=\"ml.c5.9xlarge\"\n",
    ")\n",
    "\n",
    "train_instance_count = ParameterInteger(\n",
    "    name=\"TrainingInstanceCount\",\n",
    "    default_value=1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Training Hyper-Parameters\n",
    "Note that `max_seq_length` is re-used from the processing hyper-parameters above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = ParameterInteger(\n",
    "    name=\"Epochs\",\n",
    "    default_value=1\n",
    ")\n",
    "    \n",
    "learning_rate = ParameterFloat(\n",
    "    name=\"LearningRate\",\n",
    "    default_value=0.00001\n",
    ") \n",
    "    \n",
    "epsilon = ParameterFloat(\n",
    "    name=\"Epsilon\",\n",
    "    default_value=0.00000001\n",
    ")\n",
    "        \n",
    "train_batch_size = ParameterInteger(\n",
    "    name=\"TrainBatchSize\",\n",
    "    default_value=128\n",
    ")\n",
    "    \n",
    "validation_batch_size = ParameterInteger(\n",
    "    name=\"ValidationBatchSize\",\n",
    "    default_value=128\n",
    ")\n",
    "    \n",
    "test_batch_size = ParameterInteger(\n",
    "    name=\"TestBatchSize\",\n",
    "    default_value=128\n",
    ")\n",
    "    \n",
    "train_steps_per_epoch = ParameterInteger(\n",
    "    name=\"TrainStepsPerEpoch\",\n",
    "    default_value=50\n",
    ")\n",
    "    \n",
    "validation_steps = ParameterInteger(\n",
    "    name=\"ValidationSteps\",\n",
    "    default_value=50\n",
    ")\n",
    "    \n",
    "test_steps = ParameterInteger(\n",
    "    name=\"TestSteps\",\n",
    "    default_value=50\n",
    ")\n",
    "    \n",
    "train_volume_size = ParameterInteger(\n",
    "    name=\"TrainVolumeSize\",\n",
    "    default_value=1024\n",
    ") \n",
    "    \n",
    "use_xla = ParameterString(\n",
    "    name=\"UseXLA\",\n",
    "    default_value=\"True\",\n",
    ")\n",
    "\n",
    "use_amp = ParameterString(\n",
    "    name=\"UseAMP\",\n",
    "    default_value=\"True\",\n",
    ")\n",
    "    \n",
    "freeze_bert_layer = ParameterString(\n",
    "    name=\"FreezeBERTLayer\",\n",
    "    default_value=\"False\",\n",
    ")\n",
    "\n",
    "enable_sagemaker_debugger = ParameterString(\n",
    "    name=\"EnableSageMakerDebugger\",\n",
    "    default_value=\"False\",\n",
    ")\n",
    "    \n",
    "enable_checkpointing = ParameterString(\n",
    "    name=\"EnableCheckpointing\",\n",
    "    default_value=\"False\",\n",
    ")\n",
    "\n",
    "enable_tensorboard = ParameterString(\n",
    "    name=\"EnableTensorboard\",\n",
    "    default_value=\"False\",\n",
    ")\n",
    "    \n",
    "input_mode = ParameterString(\n",
    "    name=\"InputMode\",\n",
    "    default_value=\"File\",\n",
    ")\n",
    "\n",
    "run_validation = ParameterString(\n",
    "    name=\"RunValidation\",\n",
    "    default_value=\"True\",\n",
    ")\n",
    "\n",
    "run_test = ParameterString(\n",
    "    name=\"RunTest\",\n",
    "    default_value=\"False\",\n",
    ")\n",
    "    \n",
    "run_sample_predictions = ParameterString(\n",
    "    name=\"RunSamplePredictions\",\n",
    "    default_value=\"False\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Metrics To Track Model Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics_definitions = [\n",
    "     {'Name': 'train:loss', 'Regex': 'loss: ([0-9\\\\.]+)'},\n",
    "     {'Name': 'train:accuracy', 'Regex': 'accuracy: ([0-9\\\\.]+)'},\n",
    "     {'Name': 'validation:loss', 'Regex': 'val_loss: ([0-9\\\\.]+)'},\n",
    "     {'Name': 'validation:accuracy', 'Regex': 'val_accuracy: ([0-9\\\\.]+)'},\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pygmentize src/tf_bert_reviews.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define a Training Step to Train a Model\n",
    "\n",
    "We configure an Estimator and the input dataset. A typical training script loads data from the input channels, configures training with hyperparameters, trains a model, and saves a model to `model_dir` so that it can be hosted later.\n",
    "\n",
    "We also specify the model path where the models from training will be saved.\n",
    "\n",
    "Note the `train_instance_type` parameter passed may be also used and passed into other places in the pipeline. In this case, the `train_instance_type` is passed into the estimator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow import TensorFlow\n",
    "\n",
    "estimator = TensorFlow(entry_point='tf_bert_reviews.py',\n",
    "                       source_dir='src',\n",
    "                       role=role,\n",
    "                       instance_count=train_instance_count, # Make sure you have at least this number of input files or the ShardedByS3Key distibution strategy will fail the job due to no data available\n",
    "                       instance_type=train_instance_type,\n",
    "                       volume_size=train_volume_size,                       \n",
    "                       py_version='py37',\n",
    "                       framework_version='2.3.1',\n",
    "                       hyperparameters={'epochs': epochs,\n",
    "                                        'learning_rate': learning_rate,\n",
    "                                        'epsilon': epsilon,\n",
    "                                        'train_batch_size': train_batch_size,\n",
    "                                        'validation_batch_size': validation_batch_size,\n",
    "                                        'test_batch_size': test_batch_size,                                             \n",
    "                                        'train_steps_per_epoch': train_steps_per_epoch,\n",
    "                                        'validation_steps': validation_steps,\n",
    "                                        'test_steps': test_steps,\n",
    "                                        'use_xla': use_xla,\n",
    "                                        'use_amp': use_amp,                                             \n",
    "                                        'max_seq_length': max_seq_length,\n",
    "                                        'freeze_bert_layer': freeze_bert_layer,\n",
    "                                        'enable_sagemaker_debugger': enable_sagemaker_debugger,\n",
    "                                        'enable_checkpointing': enable_checkpointing,\n",
    "                                        'enable_tensorboard': enable_tensorboard,                                        \n",
    "                                        'run_validation': run_validation,\n",
    "                                        'run_test': run_test,\n",
    "                                        'run_sample_predictions': run_sample_predictions},\n",
    "                       input_mode=input_mode,\n",
    "                       metric_definitions=metrics_definitions,\n",
    "#                       max_run=7200 # max 2 hours * 60 minutes seconds per hour * 60 seconds per minute\n",
    "                      )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we use the estimator instance to construct a `TrainingStep` as well as the `Properties` of the prior `ProcessingStep` used as input in the `TrainingStep` inputs and the code that will be executed when the pipeline invokes pipeline execution. This is very similar to an estimator's `fit` method, for those familiar with the existing Python SDK.\n",
    "\n",
    "In particular, we pass in the `S3Uri` of the `\"train\"`, `\"validation\"` and `\"test\"` output channel to the `TrainingStep`. The `properties` attribute of a Workflow step match the object model of the corresponding response of a describe call. These properties can be referenced as placeholder values and are resolved, or filled in, at runtime. For example, the `ProcessingStep` `properties` attribute matches the object model of the [DescribeProcessingJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeProcessingJob.html) response object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker.workflow.steps import TrainingStep\n",
    "\n",
    "training_step = TrainingStep(\n",
    "    name='Train',\n",
    "    estimator=estimator,\n",
    "    inputs={\n",
    "        'train': TrainingInput(\n",
    "            s3_data=processing_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                'bert-train'\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type='text/csv'\n",
    "        ),\n",
    "        'validation': TrainingInput(\n",
    "            s3_data=processing_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                'bert-validation'\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type='text/csv'\n",
    "        ),\n",
    "        'test': TrainingInput(\n",
    "            s3_data=processing_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                'bert-test'\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type='text/csv'\n",
    "        )        \n",
    "    },\n",
    ")\n",
    "\n",
    "print(training_step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Define a Training Step to Train a Model](img/pipeline-3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation Step\n",
    "\n",
    "First, we develop an evaluation script that will be specified in a Processing step that will perform the model evaluation.\n",
    "\n",
    "The evaluation script `evaluation.py` takes the trained model and the test dataset as input, and produces a JSON file containing classification evaluation metrics such as accuracy.\n",
    "\n",
    "After pipeline execution, we will examine the resulting `evaluation.json` for analysis.\n",
    "\n",
    "The evaluation script:\n",
    "\n",
    "* loads in the model\n",
    "* reads in the test data\n",
    "* issues a bunch of predictions against the test data\n",
    "* builds a classification report, including accuracy\n",
    "* saves the evaluation report to the evaluation directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create an instance of a `ScriptProcessor` processor and we use that in our `ProcessingStep`.\n",
    "\n",
    "Note the `processing_instance_type` parameter passed into the processor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "\n",
    "evaluation_processor = SKLearnProcessor(framework_version='0.23-1',\n",
    "                                        role=role,\n",
    "                                        instance_type=processing_instance_type,\n",
    "                                        instance_count=processing_instance_count,\n",
    "                                        env={'AWS_DEFAULT_REGION': region},\n",
    "                                        max_runtime_in_seconds=7200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pygmentize evaluate_model_metrics.py\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the processor instance to construct a `ProcessingStep`, along with the input and output channels and the code that will be executed when the pipeline invokes pipeline execution. This is very similar to a processor instance's `run` method, for those familiar with the existing Python SDK.\n",
    "\n",
    "The `TrainingStep` and `ProcessingStep` `properties` attribute matches the object model of the [DescribeTrainingJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeTrainingJob.html) and  [DescribeProcessingJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeProcessingJob.html) response objects, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.workflow.properties import PropertyFile\n",
    "\n",
    "evaluation_report = PropertyFile(\n",
    "    name='EvaluationReport',\n",
    "    output_name='metrics',\n",
    "    path='evaluation.json'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation_step = ProcessingStep(\n",
    "    name='EvaluateModel',\n",
    "    processor=evaluation_processor,\n",
    "    code='evaluate_model_metrics.py',\n",
    "    inputs=[\n",
    "        ProcessingInput(\n",
    "            source=training_step.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "            destination='/opt/ml/processing/input/model'\n",
    "        ),\n",
    "        ProcessingInput(\n",
    "            source=processing_step.properties.ProcessingInputs['raw-input-data'].S3Input.S3Uri,\n",
    "            destination='/opt/ml/processing/input/data'\n",
    "        )\n",
    "    ],\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name='metrics', \n",
    "                         s3_upload_mode='EndOfJob',\n",
    "                         source='/opt/ml/processing/output/metrics/'),\n",
    "    ],\n",
    "    job_arguments=[\n",
    "                   '--max-seq-length', str(max_seq_length.default_value),\n",
    "                  ],\n",
    "    property_files=[evaluation_report],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Define a Model Evaluation Step to Evaluate the Trained Model](img/pipeline-4.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.model_metrics import MetricsSource, ModelMetrics \n",
    "\n",
    "model_metrics = ModelMetrics(\n",
    "    model_statistics=MetricsSource(\n",
    "        s3_uri=\"{}/evaluation.json\".format(\n",
    "            evaluation_step.arguments[\"ProcessingOutputConfig\"][\"Outputs\"][0][\"S3Output\"][\"S3Uri\"]\n",
    "        ),\n",
    "        content_type=\"application/json\"\n",
    "    )\n",
    ")\n",
    "\n",
    "print(model_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Register Model Step\n",
    "\n",
    "![](img/pipeline-5.png)\n",
    "\n",
    "We use the estimator instance that was used for the training step to construct an instance of `RegisterModel`. The result of executing `RegisterModel` in a pipeline is a Model Package. A Model Package is a reusable model artifacts abstraction that packages all ingredients necessary for inference. Primarily, it consists of an inference specification that defines the inference image to use along with an optional model weights location.\n",
    "\n",
    "A Model Package Group is a collection of Model Packages. You can create a Model Package Group for a specific ML business problem, and you can keep adding versions/model packages into it. Typically, we expect customers to create a ModelPackageGroup for a SageMaker Workflow Pipeline so that they can keep adding versions/model packages to the group for every Workflow Pipeline run.\n",
    "\n",
    "The construction of `RegisterModel` is very similar to an estimator instance's `register` method, for those familiar with the existing Python SDK.\n",
    "\n",
    "In particular, we pass in the `S3ModelArtifacts` from the `TrainingStep`, `step_train` properties. The `TrainingStep` `properties` attribute matches the object model of the [DescribeTrainingJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeTrainingJob.html) response object.\n",
    "\n",
    "Of note, we provided a specific model package group name which we will use in the Model Registry and CI/CD work later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_approval_status = ParameterString(\n",
    "    name=\"ModelApprovalStatus\",\n",
    "    default_value=\"PendingManualApproval\"\n",
    ")\n",
    "\n",
    "deploy_instance_type = ParameterString(\n",
    "    name=\"DeployInstanceType\",\n",
    "    default_value=\"ml.m5.4xlarge\"\n",
    ")\n",
    "\n",
    "deploy_instance_count = ParameterInteger(\n",
    "    name=\"DeployInstanceCount\",\n",
    "    default_value=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_package_group_name = f\"BERT-Reviews-{timestamp}\"\n",
    "\n",
    "print(model_package_group_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_image_uri = sagemaker.image_uris.retrieve(\n",
    "    framework=\"tensorflow\",\n",
    "    region=region,\n",
    "    version=\"2.3.1\",\n",
    "    py_version=\"py37\",\n",
    "    instance_type=deploy_instance_type,\n",
    "    image_scope=\"inference\"\n",
    ")\n",
    "print(inference_image_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.workflow.step_collections import RegisterModel\n",
    "\n",
    "register_step = RegisterModel(\n",
    "    name=\"RegisterModel\",\n",
    "#    entry_point='inference.py', # Adds a Repack Step:  https://github.com/aws/sagemaker-python-sdk/blob/01c6ee3a9ec1831e935e86df58cf70bc92ed1bbe/src/sagemaker/workflow/_utils.py#L44\n",
    "#    source_dir='src',\n",
    "    estimator=estimator,\n",
    "    image_uri=inference_image_uri, # we have to specify, by default it's using training image\n",
    "    model_data=training_step.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "    content_types=[\"application/json\"],\n",
    "    response_types=[\"application/json\"],\n",
    "    inference_instances=[deploy_instance_type],\n",
    "    transform_instances=[\"ml.c5.18xlarge\"],\n",
    "    model_package_group_name=model_package_group_name,\n",
    "    approval_status=model_approval_status,\n",
    "    model_metrics=model_metrics\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Model for Deployment Step\n",
    "\n",
    "![](img/pipeline-5.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.model import Model\n",
    "\n",
    "model_name = 'bert-model-{}'.format(timestamp)\n",
    "\n",
    "model = Model(\n",
    "    name=model_name,\n",
    "    image_uri=inference_image_uri,\n",
    "    model_data=training_step.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "    sagemaker_session=sess,\n",
    "    role=role,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.inputs import CreateModelInput\n",
    "\n",
    "create_inputs = CreateModelInput(\n",
    "    instance_type=deploy_instance_type, # \"ml.m5.4xlarge\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.workflow.steps import CreateModelStep\n",
    "\n",
    "create_step = CreateModelStep(\n",
    "    name=\"CreateModel\",\n",
    "    model=model,\n",
    "    inputs=create_inputs,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define a Condition Step to Check Accuracy and Conditionally Register Model\n",
    "\n",
    "![](img/pipeline-6.png)\n",
    "\n",
    "Finally, we'd like to only register this model if the accuracy of the model, as determined by our evaluation step `step_eval`, exceeded some value. A `ConditionStep` allows for pipelines to support conditional execution in the pipeline DAG based on conditions of step properties. \n",
    "\n",
    "Below, we:\n",
    "\n",
    "* define a `ConditionGreaterThan` on the accuracy value found in the output of the evaluation step, `step_eval`.\n",
    "* use the condition in the list of conditions in a `ConditionStep`\n",
    "* pass the `RegisterModel` step collection into the `if_steps` of the `ConditionStep`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_accuracy_value = ParameterFloat(\n",
    "    name=\"MinAccuracyValue\",\n",
    "    default_value=0.01\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo\n",
    "from sagemaker.workflow.condition_step import (\n",
    "    ConditionStep,\n",
    "    JsonGet,\n",
    ")\n",
    "\n",
    "minimum_accuracy_condition = ConditionGreaterThanOrEqualTo(\n",
    "    left=JsonGet(\n",
    "        step=evaluation_step,\n",
    "        property_file=evaluation_report,\n",
    "        json_path=\"metrics.accuracy.value\",\n",
    "    ),\n",
    "    right=min_accuracy_value # accuracy\n",
    ")\n",
    "\n",
    "minimum_accuracy_condition_step = ConditionStep(\n",
    "    name=\"AccuracyCondition\",\n",
    "    conditions=[minimum_accuracy_condition],\n",
    "    if_steps=[register_step, create_step], # success, continue with model registration\n",
    "    else_steps=[], # fail, end the pipeline\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define a Pipeline of Parameters, Steps, and Conditions\n",
    "\n",
    "Let's tie it all up into a workflow pipeline so we can execute it, and even schedule it.\n",
    "\n",
    "A pipeline requires a `name`, `parameters`, and `steps`. Names must be unique within an `(account, region)` pair so we tack on the timestamp to the name.\n",
    "\n",
    "Note:\n",
    "\n",
    "* All the parameters used in the definitions must be present.\n",
    "* Steps passed into the pipeline need not be in the order of execution. The SageMaker Workflow service will resolve the _data dependency_ DAG as steps the execution complete.\n",
    "* Steps must be unique to either pipeline step list or a single condition step if/else list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.workflow.pipeline import Pipeline\n",
    "\n",
    "pipeline = Pipeline(\n",
    "    name=pipeline_name,\n",
    "    parameters=[\n",
    "        input_data,\n",
    "        processing_instance_count,\n",
    "        processing_instance_type,\n",
    "        max_seq_length,\n",
    "        balance_dataset,\n",
    "        train_split_percentage,\n",
    "        validation_split_percentage,\n",
    "        test_split_percentage,\n",
    "        feature_store_offline_prefix,\n",
    "        feature_group_name,\n",
    "        train_instance_type,\n",
    "        train_instance_count,\n",
    "        epochs,\n",
    "        learning_rate,\n",
    "        epsilon,\n",
    "        train_batch_size,\n",
    "        validation_batch_size,\n",
    "        test_batch_size,\n",
    "        train_steps_per_epoch,\n",
    "        validation_steps,\n",
    "        test_steps,\n",
    "        train_volume_size,\n",
    "        use_xla,\n",
    "        use_amp,\n",
    "        freeze_bert_layer,\n",
    "        enable_sagemaker_debugger,\n",
    "        enable_checkpointing,\n",
    "        enable_tensorboard,\n",
    "        input_mode,\n",
    "        run_validation,\n",
    "        run_test,\n",
    "        run_sample_predictions,\n",
    "        min_accuracy_value,\n",
    "        model_approval_status,\n",
    "        deploy_instance_type,\n",
    "        deploy_instance_count\n",
    "    ],\n",
    "    steps=[processing_step, training_step, evaluation_step, minimum_accuracy_condition_step],\n",
    "    sagemaker_session=sess,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine the Json of the pipeline definition that meets the SageMaker Workflow Pipeline DSL specification.\n",
    "\n",
    "By examining the definition, we're also confirming that the pipeline was well-defined, and that the parameters and step properties resolve correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "from pprint import pprint\n",
    "\n",
    "definition = json.loads(pipeline.definition())\n",
    "\n",
    "pprint(definition)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit the pipeline to SageMaker and start execution\n",
    "\n",
    "Let's submit our pipeline definition to the workflow service. The role passed in will be used by the workflow service to create all the jobs defined in the steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(experiment_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ignore the `WARNING` below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = pipeline.create(role_arn=role)\n",
    "\n",
    "pipeline_arn = response[\"PipelineArn\"]\n",
    "print(pipeline_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll start the pipeline, accepting all the default parameters.\n",
    "\n",
    "Values can also be passed into these pipeline parameters on starting of the pipeline, and will be covered later. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "execution = pipeline.start(\n",
    "    parameters=dict(\n",
    "        InputData=raw_input_data_s3_uri,\n",
    "        ProcessingInstanceCount=1,\n",
    "        ProcessingInstanceType='ml.c5.2xlarge',\n",
    "        MaxSeqLength=64,\n",
    "        BalanceDataset='True',\n",
    "        TrainSplitPercentage=0.9,\n",
    "        ValidationSplitPercentage=0.05,\n",
    "        TestSplitPercentage=0.05,\n",
    "        FeatureStoreOfflinePrefix='reviews-feature-store-'+str(timestamp),\n",
    "        FeatureGroupName='reviews-feature-group-'+str(timestamp),\n",
    "        LearningRate=0.000012,\n",
    "        TrainingInstanceType='ml.c5.9xlarge',\n",
    "        TrainingInstanceCount=1,\n",
    "        Epochs=1,\n",
    "        Epsilon=0.00000001,\n",
    "        TrainBatchSize=128,\n",
    "        ValidationBatchSize=128,\n",
    "        TestBatchSize=128,\n",
    "        TrainStepsPerEpoch=50,\n",
    "        ValidationSteps=50,\n",
    "        TestSteps=50,\n",
    "        TrainVolumeSize=1024,\n",
    "        UseXLA='True',\n",
    "        UseAMP='True',\n",
    "        FreezeBERTLayer='False',\n",
    "        EnableSageMakerDebugger='False',\n",
    "        EnableCheckpointing='False',\n",
    "        EnableTensorboard='False',\n",
    "        InputMode='File',\n",
    "        RunValidation='True',\n",
    "        RunTest='Fasle',\n",
    "        RunSamplePredictions='False', \n",
    "        MinAccuracyValue=0.01,\n",
    "        ModelApprovalStatus='PendingManualApproval', \n",
    "        DeployInstanceType='ml.m5.4xlarge',\n",
    "        DeployInstanceCount=1 \n",
    "    )\n",
    ")\n",
    "\n",
    "print(execution.arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Workflow Operations: examining and waiting for pipeline execution\n",
    "\n",
    "Now we describe execution instance and list the steps in the execution to find out more about the execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "execution_run = execution.describe()\n",
    "pprint(execution_run)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add Execution Run as Trial to Experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "execution_run_name = execution_run['PipelineExecutionDisplayName']\n",
    "print(execution_run_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_execution_arn = execution_run['PipelineExecutionArn']\n",
    "print(pipeline_execution_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# List Execution Steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "# Giving the first step time to start up\n",
    "time.sleep(30)\n",
    "\n",
    "execution.list_steps()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wait for the Pipeline to Complete\n",
    "\n",
    "# _Note: If this cell errors out with `WaiterError: Waiter PipelineExecutionComplete failed: Max attempts exceeded`, just re-run it and keep waiting._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "execution.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wait for the Pipeline ^^ Above ^^ to Complete\n",
    "\n",
    "# _Note: If this cell errors out with `WaiterError: Waiter PipelineExecutionComplete failed: Max attempts exceeded`, just re-run it and keep waiting._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can list the execution steps to check out the status and artifacts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "execution.list_steps()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examine the Evalution Metrics\n",
    "\n",
    "Examine the resulting model evaluation after the pipeline completes. Download the resulting evaluation.json file from S3 and print the report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for execution_step in reversed(execution.list_steps()):\n",
    "    if execution_step['StepName'] == 'EvaluateModel':\n",
    "        processing_job_name=execution_step['Metadata']['ProcessingJob']['Arn'].split('/')[-1]\n",
    "\n",
    "describe_evaluation_processing_job_response = sm.describe_processing_job(ProcessingJobName=processing_job_name)\n",
    "\n",
    "evaluation_metrics_s3_uri = describe_evaluation_processing_job_response['ProcessingOutputConfig']['Outputs'][0]['S3Output']['S3Uri']\n",
    "evaluation_metrics_s3_uri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "evaluation_json = sagemaker.s3.S3Downloader.read_file(\"{}/evaluation.json\".format(\n",
    "    evaluation_metrics_s3_uri\n",
    "))\n",
    "\n",
    "pprint(json.loads(evaluation_json))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download the Trained S3 Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_job_arn=None\n",
    "\n",
    "for execution_step in execution.list_steps():\n",
    "    if execution_step[\"StepName\"] == \"Train\":\n",
    "        training_job_arn = execution_step[\"Metadata\"][\"TrainingJob\"][\"Arn\"]\n",
    "        \n",
    "        print(execution_step)\n",
    "        break\n",
    "print(training_job_arn)\n",
    "        \n",
    "training_job_name = training_job_arn.split('/')[-1]\n",
    "print(training_job_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_tar_s3_uri = sm.describe_training_job(TrainingJobName=training_job_name)['ModelArtifacts']['S3ModelArtifacts']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp $model_tar_s3_uri ./\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p ./downloaded_model \n",
    "!tar -zxvf model.tar.gz -C ./downloaded_model \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!saved_model_cli show --all --dir ./downloaded_model/tensorflow/saved_model/0/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# List All Artifacts Generated By The Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processing_job_name=None\n",
    "training_job_name=None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from sagemaker.lineage.visualizer import LineageTableVisualizer\n",
    "\n",
    "viz = LineageTableVisualizer(sagemaker.session.Session())\n",
    "\n",
    "for execution_step in reversed(execution.list_steps()):\n",
    "    print(execution_step)\n",
    "    # We are doing this because there appears to be a bug of this LineageTableVisualizer handling the Processing Step\n",
    "    if execution_step['StepName'] == 'Processing':\n",
    "        processing_job_name=execution_step['Metadata']['ProcessingJob']['Arn'].split('/')[-1]\n",
    "        print(processing_job_name)\n",
    "        display(viz.show(processing_job_name=processing_job_name))\n",
    "    elif execution_step['StepName'] == 'Train':\n",
    "        training_job_name=execution_step['Metadata']['TrainingJob']['Arn'].split('/')[-1]\n",
    "        print(training_job_name)\n",
    "        display(viz.show(training_job_name=training_job_name))\n",
    "    else:\n",
    "        display(viz.show(pipeline_execution_step=execution_step))\n",
    "        time.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finalize the Experiment Run Metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# -aws-processing-job is the default name assigned by ProcessingJob\n",
    "processing_job_tc = '{}-aws-processing-job'.format(processing_job_name)\n",
    "print(processing_job_tc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = sm.associate_trial_component(\n",
    "    TrialComponentName=processing_job_tc,\n",
    "    TrialName=trial_name\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# -aws-training-job is the default name assigned by TrainingJob\n",
    "training_job_tc = '{}-aws-training-job'.format(training_job_name)\n",
    "print(training_job_tc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = sm.associate_trial_component(\n",
    "    TrialComponentName=training_job_tc,\n",
    "    TrialName=trial_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Log Additional Parameters within Trial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from smexperiments import tracker\n",
    "\n",
    "processing_job_tracker = tracker.Tracker.load(trial_component_name=processing_job_tc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processing_job_tracker.log_parameters({\n",
    "    \"balance_dataset\": str(balance_dataset), \n",
    "})\n",
    "\n",
    "# must save after logging\n",
    "processing_job_tracker.trial_component.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processing_job_tracker.log_parameters({\n",
    "    \"train_split_percentage\": str(train_split_percentage), \n",
    "})\n",
    "\n",
    "# must save after logging\n",
    "processing_job_tracker.trial_component.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processing_job_tracker.log_parameters({\n",
    "    \"validation_split_percentage\": str(validation_split_percentage), \n",
    "})\n",
    "\n",
    "# must save after logging\n",
    "processing_job_tracker.trial_component.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processing_job_tracker.log_parameters({\n",
    "    \"test_split_percentage\": str(test_split_percentage), \n",
    "})\n",
    "\n",
    "# must save after logging\n",
    "processing_job_tracker.trial_component.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processing_job_tracker.log_parameters({\n",
    "    \"max_seq_length\": str(max_seq_length), \n",
    "})\n",
    "\n",
    "# must save after logging\n",
    "processing_job_tracker.trial_component.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processing_job_tracker.log_parameters({\n",
    "    \"feature_store_offline_prefix\": str(feature_store_offline_prefix), \n",
    "})\n",
    "\n",
    "# must save after logging\n",
    "processing_job_tracker.trial_component.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processing_job_tracker.log_parameters({\n",
    "    \"feature_group_name\": str(feature_group_name), \n",
    "})\n",
    "\n",
    "# must save after logging\n",
    "processing_job_tracker.trial_component.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyze Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.analytics import ExperimentAnalytics\n",
    "\n",
    "time.sleep(30)\n",
    "\n",
    "import pandas as pd\n",
    "pd.set_option(\"max_colwidth\", 500)\n",
    "\n",
    "experiment_analytics = ExperimentAnalytics(\n",
    "    experiment_name=experiment_name,\n",
    ")\n",
    "\n",
    "experiment_analytics_df = experiment_analytics.dataframe()\n",
    "experiment_analytics_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Update Model Package Approval Status\n",
    "\n",
    "The pipeline that was executed created a Model Package version within the specified Model Package Group. Of particular note, the registration of the model/creation of the Model Package was done so with approval status as `PendingManualApproval`.\n",
    "\n",
    "As part of SageMaker Pipelines, data scientists can register the model with approved/pending manual approval as part of the CI/CD workflow.\n",
    "\n",
    "We can also approve the model using the SageMaker Studio UI or programmatically as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for execution_step in execution.list_steps():\n",
    "    if execution_step['StepName'] == 'RegisterModel':\n",
    "        model_package_arn = execution_step['Metadata']['RegisterModel']['Arn']\n",
    "        break\n",
    "print(model_package_arn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_package_update_response = sm.update_model_package(\n",
    "    ModelPackageArn=model_package_arn,\n",
    "    ModelApprovalStatus=\"Approved\",\n",
    ")\n",
    "\n",
    "print(model_package_update_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for execution_step in execution.list_steps():\n",
    "    print(execution_step['StepName'])\n",
    "    if execution_step['StepName'] == 'CreateModel':\n",
    "        model_arn = execution_step['Metadata']['Model']['Arn']\n",
    "        break\n",
    "print(model_arn)\n",
    "\n",
    "model_name = model_arn.split('/')[-1]\n",
    "print(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Model Endpoint from Model Registry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_config_name = 'bert-model-epc-{}'.format(timestamp)\n",
    "print(endpoint_config_name)\n",
    "\n",
    "create_endpoint_config_response = sm.create_endpoint_config(\n",
    "    EndpointConfigName = endpoint_config_name,\n",
    "    ProductionVariants=[{\n",
    "        'InstanceType':'ml.m5.4xlarge',\n",
    "        'InitialVariantWeight':1,\n",
    "        'InitialInstanceCount':1,\n",
    "        'ModelName': model_name,\n",
    "        'VariantName':'AllTraffic'}])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_endpoint_name = 'bert-model-ep-{}'.format(timestamp)\n",
    "print(\"EndpointName={}\".format(pipeline_endpoint_name))\n",
    "\n",
    "create_endpoint_response = sm.create_endpoint(\n",
    "    EndpointName=pipeline_endpoint_name,\n",
    "    EndpointConfigName=endpoint_config_name)\n",
    "print(create_endpoint_response['EndpointArn'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/endpoints/{}\">SageMaker REST Endpoint</a></b>'.format(region, pipeline_endpoint_name)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# _Wait Until the Endpoint is Deployed_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "while True:\n",
    "    try: \n",
    "        waiter = sm.get_waiter('endpoint_in_service')\n",
    "        print('Waiting for endpoint to be in `InService`...')\n",
    "        waiter.wait(EndpointName=pipeline_endpoint_name)\n",
    "        break;\n",
    "    except:\n",
    "        print('Waiting for endpoint...')\n",
    "        time.sleep(30)\n",
    "        \n",
    "print('Endpoint deployed.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# _Wait Until the Endpoint ^^ Above ^^ is Deployed_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# List All Artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from sagemaker.lineage.visualizer import LineageTableVisualizer\n",
    "\n",
    "viz = LineageTableVisualizer(sagemaker.session.Session())\n",
    "\n",
    "for execution_step in reversed(execution.list_steps()):\n",
    "    print(execution_step)\n",
    "    # We are doing this because there appears to be a bug of this LineageTableVisualizer handling the Processing Step\n",
    "    if execution_step['StepName'] == 'Processing':\n",
    "        processing_job_name=execution_step['Metadata']['ProcessingJob']['Arn'].split('/')[-1]\n",
    "        print(processing_job_name)\n",
    "        display(viz.show(processing_job_name=processing_job_name))\n",
    "    elif execution_step['StepName'] == 'Train':\n",
    "        training_job_name=execution_step['Metadata']['TrainingJob']['Arn'].split('/')[-1]\n",
    "        print(training_job_name)\n",
    "        display(viz.show(training_job_name=training_job_name))\n",
    "    else:\n",
    "        display(viz.show(pipeline_execution_step=execution_step))\n",
    "        time.sleep(5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test the Deployed Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from sagemaker.tensorflow.model import TensorFlowPredictor\n",
    "from sagemaker.serializers import JSONLinesSerializer\n",
    "from sagemaker.deserializers import JSONLinesDeserializer\n",
    "\n",
    "predictor = TensorFlowPredictor(endpoint_name=pipeline_endpoint_name,\n",
    "                                sagemaker_session=sess,\n",
    "                                model_name='saved_model',\n",
    "                                model_version=0,\n",
    "                                content_type='application/jsonlines',\n",
    "                                accept_type='application/jsonlines',\n",
    "                                serializer=JSONLinesSerializer(),\n",
    "                                deserializer=JSONLinesDeserializer())                                "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predict the `star_rating` with Ad Hoc `review_body` Samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = [\n",
    "    {\"features\": [\"This is great!\"]},\n",
    "    {\"features\": [\"This is bad.\"]}\n",
    "]\n",
    "\n",
    "predicted_classes = predictor.predict(inputs)\n",
    "\n",
    "for predicted_class in predicted_classes:\n",
    "    print('Predicted star_rating: {}'.format(predicted_class))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "df_reviews = pd.read_csv('./data/amazon_reviews_us_Digital_Software_v1_00.tsv.gz', \n",
    "                         delimiter='\\t', \n",
    "                         quoting=csv.QUOTE_NONE,\n",
    "                         compression='gzip')\n",
    "\n",
    "df_sample_reviews = df_reviews[['review_body', 'star_rating']].sample(n=50)\n",
    "df_sample_reviews = df_sample_reviews.reset_index(drop=True)\n",
    "df_sample_reviews.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sample_reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "def predict(review_body):\n",
    "    inputs = [\n",
    "        {\"features\": [review_body]}\n",
    "    ]\n",
    "    predicted_classes = predictor.predict(inputs)\n",
    "    return predicted_classes[0]['predicted_label']\n",
    "    \n",
    "df_sample_reviews['predicted_class'] = df_sample_reviews['review_body'].map(predict)\n",
    "df_sample_reviews.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Release Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sm.delete_endpoint(\n",
    "#      EndpointName=pipeline_endpoint_name\n",
    "# )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "\n",
    "<p><b>Shutting down your kernel for this notebook to release resources.</b></p>\n",
    "<button class=\"sm-command-button\" data-commandlinker-command=\"kernelmenu:shutdown\" style=\"display:none;\">Shutdown Kernel</button>\n",
    "\n",
    "<script>\n",
    "try {\n",
    "    els = document.getElementsByClassName(\"sm-command-button\");\n",
    "    els[0].click();\n",
    "}\n",
    "catch(err) {\n",
    "    // NoOp\n",
    "}    \n",
    "</script>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%javascript\n",
    "\n",
    "try {\n",
    "    Jupyter.notebook.save_checkpoint();\n",
    "    Jupyter.notebook.session.delete();\n",
    "}\n",
    "catch(err) {\n",
    "    // NoOp\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
